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@ Image retrieving method and apparatus. 

@ An image-retrieving system for retrieving a 
desired image from a data base. A storage 
device in the data base stores a plurality of 
images, additional informatton for identifying 
each of the plurality of images and calculated 
feature data for identifying each of the images. 
A first input means inputs additk)nal-infor- 
mation data for identifying a desired image 
stored in the data base. An additional-infor- 
mation selection means selects a first set of 
candidates as the desired image by comparing 
the input additional-information with the stored 
additional-information for each of the stored 
pluralify of images. A second input device in- 
puts an illustrative image, which is illustrative of 
the desired image, and a generation device 
generates feature data from the input illustra- 
tive image. A selection device selects as the 
desired image a one of the candidates whose 
stored feature data is nK»st similar to the gener- 
ated feature data of the input illustrative image. 
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BACKGROUND OF THE INVENTION 

Field of the Invention 

This invention relates to an image retrieving 
method and apparatus for retrieving an image from 
an Image data base. 

Description of the Related Art 

In a conventional method of retrieving a desired 
image from an image data base, additional informa- 
tion, comprising, for example, the title and the type of 
image, is retrieved using keywords. 

However, in retrieval using additional informa- 
tion, it is extremely difficult to systematically provide 
keywords for all of the stored images when the size 
of the image data base is targe and the shape of each 
image is complicated, so that it is often impossible to 
retrieve a desired image by only using keywords. 

SUMMARY OF THE INVENTION 

The present invention has been made in consid- 
eratk^n of the afc>ove-described problems. 

It Is a concern of the present invention to provide 
an image retrieving method and apparatus which can 
perform very precise image retrieval to meet a user's 
requirements. 

According to one aspect, the present invention 
provides an Image retrieving method, comprising the 
steps of selecting additional information of an adapt- 
ed image by comparing input additional information 
with stored additional inf6rmatk>n of each image stor- 
ed in a data base, generating a plurality of feature 
data from input illustrative image data, calculating the 
value of the distance t>etween each of the plurality of 
feature data and feature data of image data having 
the addittonal information selected in the selecting 
step, and selecting image data having calculated dis- 
tance values in the order of smallest distance values. 

According to another aspect, the present inven- 
tion relates to an image retrieving apparatus, com- 
prising additional-information selection means for se- 
lecting additional information of an adapted image by 
comparing input additional information with stored ad- 
ditional information of each image stored in a data 
base, generatksn means for generating a plurality of 
feature data from input illustrative image data, dis- 
tance calculation means for calculating the value of a 
distance between each of the plurality of feature data 
and feature data of image data having the additional 
information selected by the additional-information se- 
lectk>n means, and selection means for selecting inv 
age data having calculated distance values in the or- 
der of smallest distance values. 

In the above-described configuration, the addi- 
tional-information selection step selects additk>nal tn- 



fomnation of an adapted image by comparing input 
additional infonnation with stored addittonal infonma- 
tk>n of each image stored in a data base, the gener- 
ation step generates a plurality of feature data from 

5 input illustrative image data, the distance calculation 
step calculates the value of the distance between 
each of the plurality of feature data and feature data 
of image data having the additional information se- 
lected in the additonal-information selection step, and 

10 image data having calculated distance values are se- 
lected in the order of smallest distance values. 

In the other conf iguratton, the additional-informa- 
tion selection means selects addittonal information of 
an adapted image by comparing in put additional infbr- 

15 mation with stored additional information of each im- 
age stored in a data base, the generation means gen- 
erates a plurality of feature data from input illustrative 
image data, the distance calculatton means calcu- 
lates the value of the distance between each of the 

20 plurality of feature data and feature data of image 
data having the additional informatton selected by the 
additlonal-informatton selectton means, and image 
data having calculated distance values are selected in 
the order of smallest distance values. 

25 The foregoing and other advantages and fea- 

tures of the present invention will become nrore ap- 
parent from the following detailed descriptk>n of the 
preferred emt>odiment taken in conjunction with the 
accompanying drawings. 

30 

BRIEF DESCRIPTION OF THE DRAWINGS 

FIG. 1 is a block diagram illustrating the config- 
uratk>n of a retrieving apparatus according to an 

35 embodiment of the present invention; 

FIG. 2 is a diagram illustrating the concept of re- 
trieval using an iQustrative image; 
FIG. 3 is a flowchart illustrating retrieval process- 
ing using additional information; 

40 FIG. 4 is a diagram illustrating retrieval process- 

ing of the present embodiment; and 
FIG. 5 is a flowchart illustrating retrieval process- 
ing using characteristic-amounts. 

45 DETAILED DESCRIPTION OF THE PREFERRED 
EMBODIMENT 

FIG. 1 is a block diagram illustrating the config- 
uration of an image retrieving apparatus according to 

50 an emt>odin>ent of the present invention. 

In FIG. 1 , a data input unit 1 comprises, for exam- 
ple, a scanner or the like for inputting data to be stor- 
ed in a data base. Afeature calculation unit 2 con^ris- 
es, for example, a RAM (Random Access Memory), 

55 which stores, for example, a software program for ex- 
tracting various kinds of characteristic-anx>unts for 
an input image. A control unit 6 extracts various kinds 
of features for an input image by reading and execut- 
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ing the software program. 

A data storage unit 3 comprises, for example, a 
hard disk, which stores input data. A retrieval-condi- 
tion input unit 4 comprises, for example, a keyboard, 
a mouse or the like for inputting retrieval conditions 5 
for retrieving desired data. A candidate determination 
unit 5 comprises a RAM, which stores a software pro- 
gram for selecting candidate images from images 
stored in the data storage unit 3 based on the retrieval 
conditions input from the retrieval-condition input unit io 
4. A display unit 7 comprises, for example, a CRT dis- 
play for displaying Image data, an input command, 
processed data and the like. The control unit 6 com- 
prises a central processing unit for controlling the en- 
tirety of the above-described blocks, and realizes the is 
functions of the respective blocks by reading and exe- 
cuting the software program. 

The function of each of the above-described 
blocks will now be described in detail. 

The data input unit 1 includes an image input unit 20 
11, such as a scanner or the like, for Inputting Image 
data, and an addltksnal-informatlon input unit 12 for 
inputting information relating to an image, such as the 
name of the image, the date, and other necessary in- 
formation. 25 

The data storage unit 3 includes an Image-data 
storage unit 31 for storing image data input through 
the data input unit 1, an additionai-infonmation stor- 
age unit 32 for storing additk>nal information of Image 
data stored In the image-data storage unit 31 so as to 30 
correspond to the image data, and a feature storage 
unit 33 for storing a feature calculated by the feature 
calculation unit 2 so as to correspond to each image 
data. 

Image data is retrieved by the above-described 35 
configuration. 

The candidate determination unit 5 stores the 
software program, and includes a retrieval unit 51 us- 
ing additional information, a similarity calculation unit 
52 for calculating similarity using features, and a can- 40 
didate-order determination unit 53 for sorting feature 
data of a plurality of candidates in the order of higher 
similarity. 

FIG. 3 is a flowchart illustrating procedures for 
forming a data base. I.e., processing procedures from 45 
inputting of a group of image data to be retrieved until 
storing of the input data in the data storage unit 3. A 
description will now be provided for each of the steps. 

In step S1, additional information relating to an 
image to be input, comprising the name of the image, so 
the date, and other necessary informatton, is input 
through the additional-information Input unit 12. 

In step S2, image data corresponding to the ad- 
ditional information input in step S1 is input through 
the inr^ge input unit 11, comprising the scanner. 55 

In step S3, the feature calculatbn unit 2 extracts 
a plurality of features indicating characteristics of the 
corresponding image from the image data input in 



step S2. 

The feature may comprise the number of black 
pixels in each of a plurality of blocks obtained by div- 
iding an image region, the number of black pbcels on 
each line, the number of black/white reversal portions 
on each line, or a combinatton of at least one of the 
above-described numbers. 

In step S4, the additional information input in step 
SI is stored in the additionai-infonmation storage unit 
32. The additional information further comprises 
positional information relating to a position where the 
image data is to be stored in step S5, and positional 
information relating to a position where the extracted 
features are to be stored In step S6. 

In step S5, the input Image data Is stored in the 
image data storage unit 31 of the data storage unit 3. 

In step 36, the extracted features are stored in 
the feature storage unit 33. 

The data relating to the input image data can be 
easily stored so as to correlate with each other, for ex- 
ample, by using a data base management system 
comprising a correlated data base or the like. 

Next, a descriptk)n will be provided for a method 
of inputting retrieval conditions and retrieving neces- 
sary image data from a group of stored image data, 
with reference to the flowchart shown in FIG. 4. 

Three methods of retrieving a desired image are 
described below. In a first method, a desired image 
is retrieved by additional informatton in a series of 
steps starting from step S11. 

In a second method, a similar image is obtained 
from an illustrative image in a series of steps starting 
from step 812. 

In a third method, the first and second methods 
are combined. 

First, the first method will be described. 

In the first retrieval method using addittonal infor- 
mation, previously known additional information relat- 
ing to a desired image, comprising the name of the im- 
age, whether the image comprises a color Image or 
a monochromatic image, the date of registratk)n of 
the image, the type of image, or the like, is input 
through a retrieving-additional-informatbn input unit 
43, and adapted additional information is retrieved 
from additional information stored in the additional-in- 
formation storage unit 32. The retrieval by additional 
information can easily be executed using a correlat- 
ed-data-base operation language such as SQL or the 
like. 

Next, the second method will be described. 

In the second retrieval method using illustrative 
images, the user inputs a plurality of illustrative im- 
ages, which are illustrative of the desired image, 
through an Illustrative-image input unit 41 , and an im- 
age which resembles one of the illustrative images is 
retrieved. 

Image data stored in the data base may be used 
as the illustrative images. In this case, since the se- 
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lected illustrative image completely coincides with 
one of the stored image data, the coincident image 
data is selected as a first candidate. Thereafter, re- 
maining image data is sequentially selected as can- 
didates starting from a second candidate in the order 5 
of similarity. 

FIG. 2 is a schematic diagram illustrating the 
method of retrieving image data from a data base us- 
ing an illustrative image. FIG. 5 is a flowchart illus- 
trating the processing procesures of the method. io 

First, in step S30. a plurality of features are cal- 
culated by the feature calculation unit 2 using an illus- 
trative image 20 input through the illustrative-image 
input unit 41. For example, a feature calculation by 
calculation unit 2 of the illustrative image 20 shown is 
in FIG. 2 is performed, whereby a feature pO is ob- 
tained. 

In step S31, one feature is read from the disk of 
the feature storage unit 33 of the data base into a 
RAM 8 for operation. For example, in FIG. 2, a feature 20 
pi is read from the feature storage unit 33 of the data 
base. Although In the case of FIG. 2, only one feature 
Is read at one time, a plurality of features may, of 
course, be read if the capacity of the RAM 8 for op- 
eration is large. 25 

In step S32, the distance t>etween each of the 
features of the data base stored In the RAM 8 for op- 
eration and the feature of the illustrative image is cal- 
culated (see FIG. 2). Image data which has the small- 
est distance value is determined to be most similar to 30 
the illustrative image. This processing is perfonmed 
by the similarity calculation unit 52. 

In step S33, it ts determined if all features stored 
in the data base have t>een read and retrieved. If the 
result of the determination is negative, the process re- 3S 
turns to step S31 in order to repeat the calculation of 
the distance for the next data to t>e retrieved. If the re- 
sult of the determination is affirmative, the process 
proceeds to step S34. 

In step S34, features of the data base having dis- 40 
tance values smaller than a predetermined threshold 
ara sorted in the order of smallest distance values. A 
result of retrieval 21 shown in FIG. 2 illustrates an ex- 
ample of such processing. This processing is per- 
formed by the candidate-order determination unit 53. 45 

A description has been provided of the retrieval 
method by features using a single illustrative image. 
When using at least two illustrative images, the 
above-described distance is calculated for each of the 
illustrative images, the average of obtained distance so 
values is made a final distance value, sorting per- 
formed based on the average, and the order of can- 
didates is determined. Various other methods for ob- 
taining the final order of candidates can be consid- 
ered. In one method, the user assigns a weighted val- 55 
ue for the distance value for each illustrative image, 
and the final distance value is calculated using as- 
signed values. In another method, the order of candi- 



dates is determined for each illustrative image, and 
the final order of candidates is obtained from the re- 
lationship among the orders of candidates for respec- 
tive illustrative images, for example, from the sum of 
the orders of the same candidate. By using a plurality 
of illustrative images in the above-described manner, 
it is possible to increase the amount of information 
used for retrieval, and to perform image retrieval 
which is most suitable for the user's requirements. 

The selectbn of the feature to be used can be as- 
signed by an input operation by the user through a 
keyt>oard or the like. When the feature to be used is 
not assigned, the calculation of the distance is per- 
formed using a predetermined feature. 

In the present emt>odiment, similarity between 
images is obtained by calculating the distance be- 
tween the features of the respective images. 

Next, a description will be provided of the entire 
retrieval processing of the present emt>odiment with 
reference to the flowchart shown in FIG. 4. 

In step S10, one of the following three methods 
is selected through the keyt>oard, serving as the re- 
trieval-condition input unit 4. 

1 . Image retrieval is performed using additional 
information as a retrieval key. 

2. Image retrieval is performed using features as 
a retrieval key. 

3. Image retrieval is performed using both addi- 
tional information and features as a retrieval key. 
In step S11 and steps which follow step S 11, im- 
age retrieval processing is performed using addition- 
al information as a retrieval key. 

First, in step S11, matching processing between 
additional information inputthrough the retrieving-ad- 
ditional-information input unit 43 and additional infor- 
mation stored in the additional-information storage 
unit 32 of the data base are used in order to retrieve 
an image having adapted additional informatk)n. 

In step SI 4, it is determined if an image having 
adapted additional information is present 

If the result of the determination is negative, the 
process proceeds to step S20. If the result of the de- 
termination is affirmative, the image is determined to 
be a candkiate image, and the process proceeds to 
step SI 9. 

In step S12 and steps which follow step S1Z im- 
age retrieval processing is performed using features 
as a retrieval key. 

First, in step SI 2, the processing conforming to 
the flow of image retrieval by features, which has al- 
ready t>een described with reference to FIG. 5, is per- 
formed. Candidate images are sorted in the order of 
higher similarity in terms of features. 

In step SI 5, it is determined if the similarity of an 
image is greater than a predetermined threshold. 

If the result of the determination is negative, the 
process proceeds to step S20. If the result of the de- 
termination is affirmative, the image is determined to 
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be a candidate image, and the process proceeds to 
step S19. 

In step S13 and steps which follow step S13. im- 
age retrieval processing is performed using both ad- 
ditional information and features as a retrieval key. 

First, in step S13. retrieval by additional informa- 
tion, which has already been described in step S11, 
is perfomned. 

In step S16, it is determined if adapted additional 
information is present 

In step S17, only images retrieved by using addi- 
tional information as a key are retrieved by the fea- 
tures of respective images. This processing is equiv- 
alent to the processing described with reference to 
FIG. 5, and images retrieved by features are sorted 
In the order of higher similarity. 

In step S18, It is determined if the similarity of a 
retrieved Image Is greater than a predetermined 
threshold. 

If the result of the determination is negative, the 
process proceeds to step S20. If the result of the de- 
termination is affirmative, the image Is detemined to 
be a candidate image, and the process proceeds to 
step S19. 

In step S19, retrieved candidate images are dis- 
played on the picture surface of the display unit 7, and 
the process Is terminated. 

In step S20, informatk)n indicating the absence of 
candidate images is displayed on the display unit 7. 

By performing retrieval by combining Illustrative 
images and additional information in the above- 
described manner, previously known additional infor- 
mation can be added to similarity retrieval by illustra- 
tive images, so that it is possible to perform retrieval 
which is nr>08t suited to the user's requirements. 

In another method of performing similarity re- 
trieval using a plurality of illustrative images, similar- 
ity retrieval can be performed by individually assign- 
ing a feature used for calculating similarity for each 
illustrative image by the feature selection unit 42. For 
example, similarity retrieval is performed by calculat- 
ing similarity using features a and b for an illustrative 
image X, and using features c, d and e for an illustra- 
tive image Y. 

According to this method, since important fea- 
tures of each illustrative Image can be individually as- 
signed, accuracy In entire retrieval can be further im- 
proved. 

The present invention may be applied to a system 
which comprises a plurality of apparatuses, or to an 
apparatus which comprises a single unit The present 
invention may, of course, be applied when the object 
of the invention is achieved by supplying a system or 
an apparatus with a program. 

As described above, the following effects are ot>- 
tained according to the atK)ve-described embodi- 
ment. That is, in an operation of retrieving a desired 
image from an image data base, by retrieving similar 
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images from a plurality of illustrative images and ad- 
ditional information, accuracy in retrieval is improved, 
and efficient similarity retrieval can be realized. Ad- 
ditional information alone, features alone, or a com- 

5 bination of additional irrformation and features can be 
selected as a retrieval key, so that flexible Image re- 
trieval which is nK>st suitable to the user's require- 
ments can be performed. 

The individual components shown in outline or 

10 designated by blocks in the drawings are all well 
known in the image retrieving method and apparatus 
arts and their specific construction and operation are 
not critical to the operation or the best mode for car- 
rying out the invention. 

15 While the present invention has been described 

with respect to what is presently considered to be the 
preferred embodiments. It is to be understood that the 
invention is not limited to the disclosed embodiments. 
To the contrary, the present invention is intended to 

20 cover various modifications and equivalent arrange- 
ments Included within the spirit and scope of the ap- 
pended claims. The scope of the following claims is 
to be accorded the broadest interpretatton so as to en- 
compass alt such modifications and equivalent struc- 

25 tures and functions. 



Claims 

30 1. An image retrieving method, comprising the 
steps of: 

selecting additk>nal information of an 
adapted image by comparing input additional in- 
formation with additional information of each im- 
35 age stored in a data base; 

generating a plurality of feature data from 
input illustrative image data; 

calculating the value of the distance be- 
tween each of the plurality of feature data and 
40 feature data of image data having the additbnal 

information selected in the selecting step; and 

selecting image data having calculated 
distance values In the order of smallest distance 
values. 

45 

2. A method according to Claim 1 , further compris- 
ing the steps of: 

selecting image data having distance val- 
ues smaller than a predetermined value; 
so sorting the image data selected in the se- 

lecting step in the order of smallest distance val- 
ues; and 

displaying the sorted image data. 

55 3. An image retrieving apparatus, comprising: 

additional-informatbn selection means for 
selecting additional information of an adapted im- 
age by comparing input additional information 

5 
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with additional information of each image stored 
in a data base; 

generation means for generating a plural- 
ity of feature data from input illustrative image 
data; s 

distance calculation means for calculating 
the value of the distance between each of the 
plurality of feature data and feature data of image 
data having the additional information selected 
by said additional-information selection means; io 
and 

selection means for selecting image data 
having calculated distance values in the order of 
smallest distance values. 

IS 

4. An apparatus according to Claim 3, further com- 
prising: 

image-data selection means for selecting 
image data having distance values smaller than 
a predetermined value; 20 

means for sorting the image data selected 
by said image-data selection means in the order 
of smallest distance values; and 

display means for displaying the sorted 
image data. 25 

5- An image-retrieving system for retrieving a de- 
sired image from a data base, comprising: 

a storage means in said data base for stor- 
ing a plurality of images, additional-information 30 
for identifying each of the plurality of inrrages and 
calculated feature data for identifying each of the 
images; 

a first input means for inputting additional- 
information data for identifying a desired image as 
stored in said data base; 

additional-information selection means for 
selecting a set of candidates as the desired im- 
age by comparing the input additional-lnforma- 
tk>n with the stored additional-information for 40 
each of the stored plurality of images; 

a second input means for inputting an il- 
lustrative image of the desired image; 

generation means for generating feature 
data from the input Illustrative image; 4S 

similarity calculatton means for calculat- 
ing the similarity between the stored feature data 
of each of the set of candidates and the generat- 
ed feature data of the input illustrative Image; and 

selection means for selecting as the de- so 
sired image a one of the candidates whose stored 
feature data is most similar to the generated fea- 
ture data of the input illustrative image. 

6. The image retrieving system of Claim 5, wherein ss 
said similarity calculation means calculates sim- 
ilarity based on the distance between the stored 
feature data of each of the set of candidates and 



the generated feature data of the illustrative in>- 
age. 

7. The image retrieving system of Claim 5, wherein 
said selection means selects the desired image 
based on the smallest distance between the stor- 
ed feature data of each of the set of candidates 
and the generated feature data of the illustrative 
image. 

8. The image retrieving system of Claim 7, wherein 
said selection means includes: 

means for sorting the candidates based in 
the order of smallest distance between the stored 
feature data of each of the set of candidates and 
the generated feature amount of the illustrath^e 
image; and 

display means for displaying the sorted 
candidates so as user may select one of the can- 
didates as the desired image. 

9. An image-retrieving method for retrieving a de- 
sired image from a data base, comprising the 
steps of: 

storing a plurality of images, additional-in- 
formation for identifying each of the plurality of 
images and calculated feature data for identify- 
ing each of the images; 

inputting additional-information data for 
identifying a desired image stored in said data 
base; 

selecting a set of candidates as the de- 
sired image by comparing the in put additional-in- 
formation with the stored additional-information 
for each of the stored plurality of images; 

inputting an illustrative image of the de- 
sired image; 

generating feature data from the input il- 
lustrative image; 

calculating the similarity between the stor- 
ed feature data of each of the set of candidates 
and the generated feature data of the input illus- 
trath/e image; and 

selecting as the desired image a one of the 
candidates whose stored feature data is most 
similar to the generated feature data of the input 
illustrative image. 

10. The image retrieving method of Claim 9, wherein 
calculating the similarity is based on the distance 
between the stored feature data of each of the set 
of candidates and the generated feature data of 
the illustrative image. 

11. The image retrieving method of Claim 1 0, where- 
in selecting the desired image is based on the 
smallest distance between the stored feature 
data of each of the set of candidates and the gen- 
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erated feature data of the illustrative Image. 

1Z The image retrieving method of Claim 11, where- 
in the step of selecting the desired image in- 
cludes the steps of: 

sorting the candidates based in the order 
of smallest distance between the stored feature 
data of the set of candidates and the generated 
feature data of the illustrative image; and 

displaying the sorted candidates so a user 
may select one of the candidates as the desired 
image. 

13. A nnethod of retrieving stored image data com- 
prising generating feature data from input image is 
data and comparing the generated feature data 
with equivalent data of stored image data to re- 
trieve image data on the basis of the degree of 
correspondence between the compared sets of 
data. 20 

14. A method according to claim 13, wherein addi- 
tional data pertaining to data to be retrieved is 
also used to retrieve data from the stored image 
data. 25 
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